Alertmanager

Parameters

The notification service is used to push events to Alertmanager, and the following settings need to be specified:

  • targets - the alertmanager service address, array type
  • scheme - optional, default is “http”, e.g. http or https
  • apiPath - optional, default is “/api/v2/alerts”
  • insecureSkipVerify - optional, default is “false”, when scheme is https whether to skip the verification of ca
  • basicAuth - optional, server auth
  • bearerToken - optional, server auth
  • timeout - optional, the timeout in seconds used when sending alerts, default is “3 seconds”

basicAuth or bearerToken is used for authentication, you can choose one. If the two are set at the same time, basicAuth takes precedence over bearerToken.

Example

Prometheus Alertmanager config

  1. global:
  2.   resolve_timeout: 5m
  4. route:
  5.   group_by: ['alertname']
  6.   group_wait: 10s
  7.   group_interval: 10s
  8.   repeat_interval: 1h
  9.   receiver: 'default'
  10. receivers:
  11. - name: 'default'
  12.   webhook_configs:
  13.   - send_resolved: false
  14.     url: 'http://10.5.39.39:10080/api/alerts/webhook'

You should turn off “send_resolved” or you will receive unnecessary recovery notifications after “resolve_timeout”.

Send one alertmanager without auth

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: <config-map-name>
  5. data:
  6.   service.alertmanager: |
  7.     targets:
  8.     - 10.5.39.39:9093

Send alertmanager cluster with custom api path

If your alertmanager has changed the default api, you can customize “apiPath”.

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: <config-map-name>
  5. data:
  6.   service.alertmanager: |
  7.     targets:
  8.     - 10.5.39.39:443
  9.     scheme: https
  10.     apiPath: /api/events
  11.     insecureSkipVerify: true

Send high availability alertmanager with auth

Store auth token in argocd-notifications-secret Secret and use configure in argocd-notifications-cm ConfigMap.

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: <secret-name>
  5. stringData:
  6.   alertmanager-username: <username>
  7.   alertmanager-password: <password>
  8.   alertmanager-bearer-token: <token>
  • with basicAuth
  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: <config-map-name>
  5. data:
  6.   service.alertmanager: |
  7.     targets:
  8.     - 10.5.39.39:19093
  9.     - 10.5.39.39:29093
  10.     - 10.5.39.39:39093
  11.     scheme: https
  12.     apiPath: /api/v2/alerts
  13.     insecureSkipVerify: true
  14.     basicAuth:
  15.       username: $alertmanager-username
  16.       password: $alertmanager-password
  • with bearerToken
  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: <config-map-name>
  5. data:
  6.   service.alertmanager: |
  7.     targets:
  8.     - 10.5.39.39:19093
  9.     - 10.5.39.39:29093
  10.     - 10.5.39.39:39093
  11.     scheme: https
  12.     apiPath: /api/v2/alerts
  13.     insecureSkipVerify: true
  14.     bearerToken: $alertmanager-bearer-token

Templates

  • labels - at least one label pair required, implement different notification strategies according to alertmanager routing
  • annotations - optional, specifies a set of information labels, which can be used to store longer additional information, but only for display
  • generatorURL - optional, default is ‘{{.app.spec.source.repoURL}}’, backlink used to identify the entity that caused this alert in the client

the label or annotations or generatorURL values can be templated.

  1. context: |
  2.   argocdUrl: https://example.com/argocd
  4. template.app-deployed: |
  5.   message: Application {{.app.metadata.name}} has been healthy.
  6.   alertmanager:
  7.     labels:
  8.       fault_priority: "P5"
  9.       event_bucket: "deploy"
  10.       event_status: "succeed"
  11.       recipient: "{{.recipient}}"
  12.     annotations:
  13.       application: '<a href="{{.context.argocdUrl}}/applications/{{.app.metadata.name}}">{{.app.metadata.name}}</a>'
  14.       author: "{{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}"
  15.       message: "{{(call .repo.GetCommitMetadata .app.status.sync.revision).Message}}"

You can do targeted push on Alertmanager according to labels.

  1. template.app-deployed: |
  2.   message: Application {{.app.metadata.name}} has been healthy.
  3.   alertmanager:
  4.     labels:
  5.       alertname: app-deployed
  6.       fault_priority: "P5"
  7.       event_bucket: "deploy"

There is a special label alertname. If you don’t set its value, it will be equal to the template name by default.